{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using Python for Information Retrieval\n",
    "\n",
    "In this unit, we'll use Python to turn a bunch of loose text documents into a real-life database. (Note: This database was created for a project by R. Terman and E. Voeten, and was processed using much the same process as you'll be learning here.)\n",
    "\n",
    "The lecture and problem set will leverage your new Python skills, especially working with text, lists, and dictionaries; writing for-loops, conditional statements, and functions; and thinking like a programmer.\n",
    "\n",
    "**About the Data**\n",
    "\n",
    "We'll be creating a database from [Universal Period Review outcome reports](http://www.ohchr.org/EN/HRBodies/UPR/Pages/BasicFacts.aspx).\n",
    "\n",
    "The Universal Periodic Review (UPR) is a process run by the United Nations Human Rights Council, which involves a periodic review of the human rights records of all 193 UN Member States.\n",
    "\n",
    "Reviews take place through an interactive discussion between the State under review and other UN Member States. During this discussion any UN Member State can pose questions, comments and/or make recommendations to the States under review. States under review can then respond, stating which recommendations they reject, accept, will consider, etc. Reports are then drawn up detailing this discussion.\n",
    "\n",
    "We will be analyzing outcome reports from the 2014 Universal Period Reviews of 42 countries, which we retrieved [here](http://www.ohchr.org/EN/HRBodies/UPR/Pages/Documentation.aspx) and formatted as text documents.\n",
    "\n",
    "The goal is to convert these semi-structured texts to a tabular dataset of **recommendations** with the following variables:\n",
    "\n",
    "1. Text of recommendation (*text*)\n",
    "2. Country to which the recommendation is directed (*to*)\n",
    "3. Country that is making the recommendation (*from*)\n",
    "4. The year when the review took place (*year*)\n",
    "5. The response to the recommendation, i.e. whether the reviewed country rejects, accepts, etc (*decision*)\n",
    "\n",
    "In other words, we want to turn this:\n",
    "\n",
    "<img src=\"img/text.png\" width=\"600\">\n",
    "\n",
    "into this:\n",
    "\n",
    "<img src=\"img/tabular.png\" width=\"400\">\n",
    "\n",
    "Take a few minutes to look at the files, which are located in `data/txts`, and get a sense for how they're structured."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PART A: Start with one document"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Read, Clean, Assign\n",
    "\n",
    "We're going to start off working with just one document. Then we'll be able to put that into a loop that runs on all the documents.\n",
    "\n",
    "**task**:\n",
    "\n",
    "1. Read one document\n",
    "2. Collect information on the country and year\n",
    "3. Keep the section we're interested in\n",
    "4. Turn each line into an item in a list.\n",
    "\n",
    "**skills**:\n",
    "- file reading\n",
    "- [string](https://github.com/dlab-berkeley/python-intensive/blob/master/Glossary.md#string) slicing\n",
    "- string methods\n",
    "- indexing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 Read in \"cotedivoire2014.txt\"\n",
    "\n",
    "Fill in the blanks to read in the file. We'll need to include the `encoding='utf8'` optional parameter to the `open()` function to ensure that the text file is read correctly on all operating systems."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "directory = './data/txts'\n",
    "file_name = \"cotedivoire2014.txt\"\n",
    "# FILL ME OUT"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 Assign country and year variables \n",
    "\n",
    "You'll notice that the file name consists of the name of the country and the year. We can use this to get that information. Slice the file name to create 2 new variables, `country`, and `year`.\n",
    "\n",
    "Be careful! Remember that we are going to apply this to the other file names later. Make sure that however you slice \"cotedivoire2014.txt\" would work for the other files in the `data/txts` directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# FILL ME OUT"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3 Get the Recommendations Section\n",
    "\n",
    "Note that the section we want starts with `\"II. Conclusions and/or recommendations\\n\"`. What [method](https://github.com/dlab-berkeley/python-intensive/blob/master/Glossary.md#method) would you use to get everything after this substring? Fill in the blank below and [assign](https://github.com/dlab-berkeley/python-intensive/blob/master/Glossary.md#assign) the value to a new variable called `rec_text`.\n",
    "\n",
    "Note: there is certainly more than one way to do this, but the code below suggests one string method in particular. If you have time, think about what other methods or libraries you could use to get certain substrings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# FILL ME OUT"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.4 Turn it into a list\n",
    "\n",
    "Using a string method, turn the string above into a list of lines, and store it in a variable called `recs`. Remember that a new line is represented by `\\n`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# FILL ME OUT"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.5 Make a function\n",
    "\n",
    "Let's put all of that code into a function that will read in a file and return a list of recommendations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def read_recommendations(filename):\n",
    "    # ADD YOUR CODE FROM SECTION 1 HERE    \n",
    "    return recs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Chunk Recommendations\n",
    "\n",
    "**task**:\n",
    "\n",
    "These texts have 3 sections each. \n",
    "1. The first section contains those recommendations the country supports. \n",
    "2. The second section contains recommendations the country will examine. \n",
    "3. The third contains recommendations the country explicitely rejects. \n",
    "\n",
    "We want to chunk the the text into three lists, `accept`, `examine`, `reject` -- each containing their respective recommendations.\n",
    "\n",
    "**skills**:\n",
    "- string methods\n",
    "- list comprehensions\n",
    "- conditionals\n",
    "- indexing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1: Find the paragraph numbers\n",
    "\n",
    "Each section starts with a main paragraph number (e.g. **123**). The individual recommendations are then noted as subparagraphs (e.g. **123.1, 123.2** etc.).\n",
    "\n",
    "All the accepted recommendations have the same main paragraph number (**123**). Next come the recommendations which will be examined, whose main paragraph number is just the next integer (**124**). After that are the rejected recommendations, with the next integer as their main paragraph number (**125**).\n",
    "\n",
    "We can't know the paragraph numbers beforehand. But we *can* leverage our knowledge of the structure of the documents to get them.\n",
    "\n",
    "Fill in the blanks below to create 3 variables containing the 3 paragraph numbers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# FILL ME OUT"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 Parse the text\n",
    "\n",
    "Now create 3 new lists: `accept`, `examine`, `reject.` Complete the for loop code to filter through `recs` and assign each recommendation to its corresponding section.\n",
    "\n",
    "**hint**: How do you know if a line belongs to a section? It starts with the main paragraph number for that section. So use the **.startswith()** method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# allocate lists for the 3 types of recommendations\n",
    "accept_recs = []\n",
    "examine_recs = []\n",
    "reject_recs = []\n",
    "\n",
    "# FILL ME OUT"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 Make a function\n",
    "\n",
    "Let's again put the code we just created to parse the text into 3 separate lists into a function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def parse_recommendations(recs):\n",
    "    # PUT YOUR CODE HERE FROM SECTION 2\n",
    "    \n",
    "    # Put the three lists of recommendations into a tuple so it can be returned\n",
    "    return (accept_recs, examine_recs, reject_recs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Get Recommending Country\n",
    "\n",
    "**skills**\n",
    "\n",
    "- string methods\n",
    "- indexing\n",
    "- functions\n",
    "\n",
    "**task**\n",
    "- extract the substring representing the recommending country."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 Extracting the Country\n",
    "\n",
    "Take a look at several recommendations to get an idea of their format. I've given you several samples below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for cur_rec in accept_recs[:5]: \n",
    "    print(cur_rec)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that they're all formatted the same way, with the recommending country in parenthesis at the end, in between parentheses.\n",
    "\n",
    "Using your string skills, find a way to pull out the recommending country from the first recommendation (stored in `first_rec` below)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "first_rec = accept_recs[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# FILL ME OUT"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 Create a Function\n",
    "\n",
    "Using the code you just wrote, create a function called `get_country` that passes an individual recommendation and returns the recommending country"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_country(rec):\n",
    "    # YOUR CODE HERE\n",
    "    return(rec_country)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# test your code\n",
    "get_country(first_rec)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Processing all Recommendations\n",
    "\n",
    "**task**:\n",
    "\n",
    "We now want to create a new list for each variable we eventually want in our output csv file. Each list will contain a single value per individual recommendation. The five variables we want a list for are: \n",
    "\n",
    "1. `to`: the country under review\n",
    "2. `from`: the country (or countries) giving the recommendation\n",
    "3. `year`: the year of the review (all 2014 here)\n",
    "4. `decision`: whether the recommendation was supported, rejected, etc.\n",
    "5. `text`: the text of the recommendation\n",
    "\n",
    "To make it easier to store these data (and later to write it out to a csv file), we'll create a dictionary with an empty list for each of these variable names.\n",
    "\n",
    "**skills**:\n",
    "- loops\n",
    "- dictionaries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rec_output = {'to':[],\n",
    "              'from':[],\n",
    "              'year':[],\n",
    "              'decision':[],\n",
    "              'text':[]}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1 Process the `accept` Recommendations\n",
    "\n",
    "The code below loops through all the recommentations in the `accept` list and appends an item to each of the 5 lists within the dictionary defined above. Fill in the blanks to complete the code.\n",
    "\n",
    "(Remember we've already created the `country` and `year` variables above!)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# FILL ME OUT"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2 Make a function \n",
    "\n",
    "Now write a function that does the same for any list of recommendations. It should first create an output dictionary and then populate that dictionary. Think about all the parameters that the function should take in order to fill in all 5 fields of the `rec_output` dictionary. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def process_recs(recs, to_country, year, decision_type):\n",
    "    # YOUR CODE FROM SECTION 4.1 HERE, UPDATED TO USE THE PARAMETERS PASSED IN\n",
    "    ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### 4.3 Process all the Recommendations\n",
    "\n",
    "Now use the function that you just wrote to process the recommendations from the `accept` the `examine` and `reject` recommendation lists."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# FILL ME OUT\n",
    "\n",
    "# uncomment to test your code\n",
    "# print(len(rec_output['to']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " ### 4.4 Combine output dictionaries\n",
    " \n",
    "Now let's write a function that takes a list of output recommendation dictionaries and creates a new one that is the combination of all of them. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def combine_outputs(dicts):\n",
    "    # FILL ME OUT\n",
    "    return output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now combine the output dictionaries for the accept, examine, and reject recommendations into a single output dictionary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# FILL IN THE BELOW LINE TO USE THE combine_outputs FUNCTION FROM ABOVE\n",
    "output_recs = ...\n",
    "\n",
    "# uncomment to test your code\n",
    "# print(len(accept_recs) + len(examine_recs) + len(reject_recs))\n",
    "# print(len(output_recs['to']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PART B: Repeat for all documents\n",
    "\n",
    "We just wrote code that takes one document and turns it into a dataset!\n",
    "\n",
    "The problem is we have 11 documents!\n",
    "\n",
    "We'll now combine the code we've written so far to create a function that can read one document at a time, and then read all 11 documents into a single dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Make a function\n",
    "\n",
    "**task**\n",
    "\n",
    "Combine the functions that you wrote above to create a single function that takes a filename as a parameter and returns a dictionary of lists representing all of the recommendations in that document.\n",
    "\n",
    "**skills**\n",
    "- Functions\n",
    "- Copying and pasting :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# complete the code below.\n",
    "def process_document(filename):\n",
    "\n",
    "    # FILL USING THE FUNCTIONS YOU'VE WRITTEN IN SECTIONS 1-4\n",
    "    \n",
    "    return(output_recs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# test your code!\n",
    "print(len(process_document(\"tuvalu2013.txt\")['to']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Process all of the files\n",
    "\n",
    "**task**\n",
    "\n",
    "1. Find the file_names in our directory.\n",
    "2. Apply the function above to all the filenames\n",
    "3. Create a master dataset\n",
    "\n",
    "**skills**\n",
    "- I/O\n",
    "- Loops\n",
    "- Functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.1 Make a list of file_names\n",
    "\n",
    "The program below reads all the file_names in the directory `data/txts`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "directory = 'data/txts'\n",
    "for file_name in os.listdir(directory):\n",
    "    print(file_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Modify the program to include only the file_names that end in `.txt` by using a string method.\n",
    "\n",
    "**hint:** We used the `.startswith()` method earlier. What do you think could work here?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6.2 Process all the documents\n",
    "\n",
    "Fill in the blanks below to process all the documents.\n",
    "\n",
    "In the last line we put the recommendations from one document into a list called `output_recs` which will hold the recommendations for all of the documents. We then need to combine all the output dictionaries from all the documents. We've written a function to do this already, which one was it? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "output_recs = []\n",
    "# FILL ME OUT"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Should be 1709\n",
    "len(output_recs_final['to'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6.3 Save to file\n",
    "\n",
    "Now we'll create a `pandas` `DataFrame` around our dataset and write it to a CSV file, and we're done!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# FILL ME OUT"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
